Why Search Engines are Used Increasingly to Offload Queries from Databases

The development of future search engine technology is no longer limited to free text. Rather, the aim is to build core indexing services that focus on extreme performance and scalability for retrieval and analysis across structured and unstructured data sources alike. In addition, binary query evaluation is being replaced with advanced frameworks that provide both fuzzy matching and ranking schemes, to separate value from noise. As another trend, analytical applications are being enabled by the computation of contextual concept relationships across billions of documents/records on-the-fly. Based on these developments in search engine technology, a set of new information retrieval infrastructure patterns are appearing: 1. the mirroring of DB content into a search engine in order to improve query capacity and user experience, 2. the use of search engine technology as the default access pattern to both structured and unstructured data in applications such as CRM and storage and document management, and 3. a paradigm shift is predicted in business intelligence. The presentation will review key trends from search engine development and relate these to concrete user scenarios. About the Speaker Bjørn Olstad is the CTO in FAST Search & Transfer and an adjunct professor at the Norwegian University of Science and Technology (NTNU). FAST has emerged as the leading provider of Enterprise Search Platforms (ESP). The FAST ESP platform has been embedded as the information access layer in applications such as Siebel, EMC Storage and Documentum. Companies like ReedElsevier, IBM, Dell, AOL, Factiva and Reuters use FAST ESP to power information retrieval and analytics solutions. Before joining FAST Olstad has been a professor at NTNU and headed development at GE Healthcare, Cardiac Ultrasound. Bjørn Olstad has published more than 70 research papers and he has been granted more than 30 patents. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the VLDB copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To copy otherwise, or to republish, requires a fee and/or special permission from the Endowment Proceedings of the 31 VLDB Conference, Trondheim, Norway, 2005